---
sidebar_position: 3
---

import AuthorDetails from '@site/src/components/AuthorDetails';

# Creating a Custom Local Model (LM) Client

DSPy provides you with multiple LM clients that you can use to execute any of your pipelines. However, if you have an API or LM that is unable to be executed by any of the existing clients hosted in DSPy, you can create one yourself!! It's not too difficult so let's see how!!

## Format of LM Client

An LM client needs to implement 3 methods at minimum: `__init__`, `basic_request` and `__call__`. So your custom LM client should follow the template below:

```python
from dsp import LM


class CustomLMClient(LM):
    def __init__(self):
        self.provider = "default"

        self.history = []

    def basic_request(self, prompt, **kwargs):
        pass

    def __call__(self, prompt, only_completed=True, return_sorted=False, **kwargs):
        pass
```

While you can mostly add to and customize the client per your requirements, the client should be configurable with some key components to utilize every feature of DSPy without interruption. One of these key features is viewing the history of calls made to the LM via `inspect_history`. To elaborate:

* `__init__`: Should contain the `self.provider="default` and `self.history=[]`. `self.history` will contain the prompt-completion pair created via LM call since the object was initialized. `self.provider` is used in `inspect_history` method and for most part you can leave it as **"default"**.
* `basic_request`: This function makes the call to the LM and retrieves the completion for the given prompt over the given `kwargs` which usually have parameters like `temperature`, `max_tokens`, etc. After you receive the completion from the LM, you must update the `self.history` list by appending the dictionary `{"prompt": prompt, "response": response}` to it. These fields are mandatory but you can add any other parameters as you see fit.
* `__call__`: This function should return the list of completions returned by the model. This can be a list of string completions in the basic case. Or it can be a tuple pair with the completion and corresponding likelihood as returned by the `Cohere` LM client. Additionally, the completion should be received by the `request` call which unless modified just calls `basic_request` as is, ensuring the history is updated as well. 

By now you must've realized the reason we have these rules is mainly for making the history inspection and modules work without breaking.

:::info
You can mitigate issues with updating the history in the `__call__` itself. So if you can take care of history updates in `__call__` itself, you just need to implement `__init__` and `__call__`.

Or if you are up for it, feel free to rewrite `inspect_history` method as per your requirements!
:::

## Implementing our Custom LM

Based on whatever we have learned so far, let's implement our custom LM that calls the Claude API. In Claude, we need to initialize 2 components to make a successful call: `API_KEY` and `BASE_URL`. The base URL from on the [**Anthropic docs**](https://docs.anthropic.com/claude/reference/messages_post) is `https://api.anthropic.com/v1/messages`. Let's write our `__init__` method:

```python
def __init__(model, api_key):
    self.model = model
    self.api_key = api_key
    self.provider = "default"
    self.history = []

    self.base_url = "https://api.anthropic.com/v1/messages"
```

Based on the implementation above, we want to now pass in our `model` like `claude-2` and the `api_key` which you'll see in Claude's API Console. Now it's time to define the `basic_request` method where we'll make the call to `self.base_url` and get the completion:

```python
def basic_request(self, prompt: str, **kwargs):
    headers = {
        "x-api-key": self.api_key,
        "anthropic-version": "2023-06-01",
        "anthropic-beta": "messages-2023-12-15",
        "content-type": "application/json"
    }

    data = {
        **kwargs,
        "model": self.model,
        "messages": [
            {"role": "user", "content": prompt}
        ]
    }

    response = requests.post(self.base_url, headers=headers, json=data)
    response = response.json()

    self.history.append({
        "prompt": prompt,
        "response": response,
        "kwargs": kwargs,
    })
    return response
```

Now it's time to define `__call__` method that'll bring it all together:

```python
def __call__(self, prompt, only_completed=True, return_sorted=False, **kwargs):
    response = self.request(prompt, **kwargs)

    completions = [result["text"] for result in response["content"]]

    return completions
```

To write it all in a single class, we'll get:

```python
from dsp import LM

class Claude(LM):
    def __init__(model, api_key):
        self.model = model
        self.api_key = api_key
        self.provider = "default"

        self.base_url = "https://api.anthropic.com/v1/messages"

    def basic_request(self, prompt: str, **kwargs):
        headers = {
            "x-api-key": self.api_key,
            "anthropic-version": "2023-06-01",
            "anthropic-beta": "messages-2023-12-15",
            "content-type": "application/json"
        }

        data = {
            **kwargs,
            "model": self.model,
            "messages": [
                {"role": "user", "content": prompt}
            ]
        }

        response = requests.post(self.base_url, headers=headers, json=data)
        response = response.json()

        self.history.append({
            "prompt": prompt,
            "response": response,
            "kwargs": kwargs,
        })
        return response

    def __call__(self, prompt, only_completed=True, return_sorted=False, **kwargs):
        response = self.request(prompt, **kwargs)

        completions = [result["text"] for result in response["content"]]

        return completions
```

That's it! Now we can configure this as an `lm` in DSPy and use it in the pipeline like any other LM Client:

```python
import dspy

claude = Claude(model='claude-2')

dspy.settings.configure(lm=claude)
```

***

<AuthorDetails name="Arnav Singhvi"/>